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PREFACE 


This  technical  report  reviews  the  progress  of  the  Armstrong 
Laboratory  psychophysiological  workload  effort  in  the  Human 
Engineering  Division  at  Wright -Patterson  Air  Force  Base.  This 
summary  covers  the  work  of  the  past  several  years  and  encompasses 
both  basic  and  applied  research  efforts.  A  broad  based  approach 
has  been  used  to  address  the  questions  of  understanding  and 
measurement  of  operator  workload  using  psychophysiological 
measures.  Heart,  eye,  respiration  and  brain  measures  have  been 
used  to  determine  the  best  use  of  each  one.  Further,  measures 
have  been  combined  to  take  advantage  of  the  strengths  of  each  one 
when  used  in  combination. 

Considerable  progress  has  been  made  by  this  effort  in  both 
the  laboratory  and  flight  environments.  The  studies  reported  here 
have  increased  our  understanding  of  operator  workload  and  the 
relationship  of  the  various  physiological  measures  to  operator 
state.  Physiological  measures  are  maturing  to  the  point  where 
they  are  being  used  by  the  design  and  test  and  evaluation 


communities.  In  addition  to  publications,  a  second  generation 


psychophysiological  test  battery  has  been  developed  for  use  by 
other  investigators  who  wish  to  employ  these  methods  in  their  own 
work.  The  development  of  the  test  battery  supports  our  goal  of 
increasing  the  use  of  physiological  measures  in  the  applied 
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INTRODUCTION 


The  measurement  and  monitoring  of  operator  state  is 
becoming  more  important  as  the  systems  that  humans  are  required 
to  operate  become  more  complex  and  place  greater  demands  on  the 
operator,  on  one  hand,  and  are  more  automated  and  boring  on  the 
other.  Airlines  currently  schedule  commercial  passenger 
flights  of  14  hours  or  more  on  highly  automated  aircraft  which 
primarily  requires  monitoring  by  the  crew.  The  modern  fighter 
aircraft  with  its  multifunction  displays  can  present  an 
overwhelming  amount  of  information  to  the  pilot,  and  its 
complex  control  and  weapons  systems  require  a  taxing  array  of 
information  processing  and  decision  options.  In  these 
environments,  workload  and  fatigue  are  important  factors  and  we 
must  understand  these  phenomena  and  derive  measures  that  allow 
us  to  monitor  them  so  that  dangerous  situations  can  be  avoided. 
This  is  true  of  high  and  low  workload  situations,  as  well  as 
fatigue  and  boredom  conditions.  All  of  these  states  can  result 
in  serious  consequences  if  they  are  not  detected  in  time  to 
take  corrective  actions.  The  same  understanding  of  the 
phenomena  and  requirement  for  measurement  tools  are  also  needed 
in  design,  display  evaluation,  simulation,  and  operational  test 
and  evaluation. 

Traditionally,  three  approaches  have  been  taken  to  measure 
operator  workload  and  state;  (1)  performance,  (2)  subjective 
and  (3)  physiological  measures.  Each  method  has  its  strengths 
and  weaknesses  and  a  combination  will  no  doubt  constitute  the 
best  approach.  Performance  measures,  when  available,  can 
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produce  large  quantities  of  data  whose  use  is  problematic  since 
there  is  no  agreement  about  which  measures  are  best.  Often 
direct  performance  measures  cannot  be  made,  especially  in  the 
case  of  operational  aircraft  that  are  not  instrumented  to 
provide  the  relevant  data.  Even  when  data  are  available,  the 
large  number  of  correct  ways  to  accomplish  a  given  mission  make 
it  difficult  to  know  how  to  score  performance  data.  Further, 
under  extremes  of  high  or  low  workload  conditions,  the  operator 
may  fail  to  respond  at  all.  Secondary  task  methodology  has 
been  used  to  gather  performance  data,  but  the  secondary  task 
may  interfere  with  primary  task  performance  or  its  performance 
may  be  omitted  under  extremes  of  primary  task  workload. 
Subjective  techniques  are  used  a  great  deal,  exclusively  in 
some  situations,  and  provide  useful  data.  However,  these  data 
are  subject  to  errors  of  memory  and  can  be  influenced  by 
operator  bias.  If  the  subjective  evaluations  are  collected 
during  task  performance,  they  are  susceptible  to  the  same 
problems  of  intrusion  as  the  secondary  task  techniques. 

Physiological  data  can  be  collected  non-intrusively  in 
terms  of  the  primary  task  since  the  physiological  activity  is 
ongoing  simultaneously  with  the  performance  of  the  task. 
Physiological  data  can  be  collected  continuously  making  it 
possible  to  monitor  subject  state  in  rapidly  changing 
environments  and  to  reduce  the  problem  of  missing  important 
episodes  due  to  lapses  in  data  collection.  Physiological  data 
recording  does  require  specialized  equipment,  but  the  current 
state-of-the-art  in  electronic  hardware  and  computer  software 
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makes  this  a  relatively  minor  problem.  Other  physiological 
signals  can  interfere  with  those  of  interest,  e.g.,  eye  blinks 
intruding  onto  EEC,  but  there  are  methods  available  to  deal 
with  these  problems. 

The  psychophysiological  laboratory  of  the  Armstrong 
Laboratory,  Workload  and  Ergonomics  Branch  has  made 
considerable  progress  in  the  past  several  years.  The  goal  of 
our  efforts  has  been  to  provide  a  better  understanding  of 
operator  state  by  using  physiological  measures  and  to  develop  a 
database  of  both  laboratory  and  field  data  so  that  on-line 
measurement  of  operator  state  can  be  accomplished  during  normal 
duty.  This  report  covers  progress  over  the  past  several  years 
and  discusses  future  directions.  Our  efforts  encompass  a  wide 
range  of  research  and  application  areas.  The  field  efforts  are 
directed  towards  utilization  of  psychophysiological  methods  to 
evaluate  operator  workload  and  state.  Our  laboratory  work  is 
aimed  at  developing  new  psychophysiological  methods,  validating 
currently  available  methods  for  use  in  "real  world"  situations, 
and  providing  new  knowledge  upon  which  to  base  future  work.  The 
goal  of  our  basic  research  effort  in  the  MEG  and  EEG 
laboratories  is  to  increase  the  knowledge  base  of  how  the  brain 
works  and  to  better  understand  human  information  processing, 
especially  during  cognition.  Progress  in  each  of  the  major 
areas  will  be  presented  and  possible  future  directions  will  be 
discussed.  To  summarize  our  activities  the  most  important 
results  will  be  highlighted. 


3 


CURRENT  PROGRESS 


FIELD  RESEARCH 

Data  have  been  collected  and  analyzed  from  crew  members 
while  flying  three  different  types  ^  aircraft,  A-7,  B-1  and  F-4 , 
These  aircraft  provide  varied  environments  which  include  single 
seat,  low  level  attack  aircraft  (A-7),  fighter-bomber  (F-4)  and 
multicrew,  low-level  bombers  (B-1) . 

A-7  Data 

The  A-7  study  afforded  us  the  opportunity  to  use 
physiological  data  to  better  understand  the  workload 
experienced  by  single-seat  aircraft  pilots  during  different 
segments  of  training  missions.  Heart  rate  and  eye  blink  data 
were  recorded  while  the  pilots  flew  in  three  different  flight 
positions.  Each  pilot  flew  in  the  lead  and  wing  positions  of 
four  ship  training  missions  and  they  also  flew  the  same  mission 
in  a  simulator.  This  unique  study  provided,  for  the  first 
time,  data  from  the  same  pilots  in  these  different  roles.  To 
our  knowledge  flight  and  simulator  data  recorded  from  the  same 
pilots  have  not  been  reported  previously.  These  data  show  the 
effects  of  the  varying  workload  associated  with  the  mission  and 
also  the  demands  of  the  three  different  roles.  Figure  1 
illustrates  the  mean  heart  rate  data  and  shows  the  effects  of 
the  different  mission  segments  (e.g.,  takeoff,  range,  cruise) 
and  the  differences  between  flying  lead,  wing  or  in  the 
simulator.  Flight  segments  with  higher  workload  were 
associated  with  higher  heart  rates  e.g.,  takeoff,  bombing  range 
and  landing.  The  simulator  data  did  not  produce  significant 
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effects  on  the  pilots'  heart  r.'^tes  across  the  mission  segments. 
With  regard  to  the  effects  of  flight  position,  the  lead 
position  demonstrated  the  highest  heart  rates,  with  the  wing 
next,  and  the  simulator  flight  lowest.  The  pilots'  blink  rates 


MISSION  EVENTS 

lead  WING  SIM 


Figure  1.  Mean  heart  rate  for  the  eight  A-7  pilots  when 
they  flew  lead,  wing  or  simulated  air-to-ground  missions .  The 
highest  heart  rates  were  during  the  missions  when  they  flew 
lead  with  wing  next  highest  and  simulation  lowest.  Note  the 
relative  lack  of  change  due  to  mission  segment  in  the  simulated 
mission.  The  abbreviations  for  the  mission  segments  represent 
the  following:  B  -  pref light  briefing,  PT  -  pre  takeoff  prior 
to  aircraft  roll,  T  -  takeoff,  LL  -  low  level  flight,  FI  to  F4 
-  inflight  baselines,  J,  S  and  9  -  G  maneuvers,  WD  -  weapons 
delivery  at  the  bombing  range,  CR  -  cruise  back  to  base,  LA  - 
landing. 
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were  significantly  reduced  by  flying  in  the  wing  position  which 
requires  greater  overall  visual  attent ’ on  since  one's  position 
relative  to  the  lead  aircraft  must  bt  ’  aintained  in  addition  to 
the  other  requirements  of  flight  (Wilson,  et  al,  1987) . 

Serendipity  provided  us  with  another  unique  and  valuable 
set  of  data.  Two  of  the  pilots  experienced  inflight 
emergencies  and  two  other  pilots  experienced  emergencies  during 
the  simulated  flight.  These  data  permit  us  to  examine  pilots' 
physiological  responses  to  potentially  life  threatening  events 
and  allowed  us  to  compare  these  responses  with  those  during 
normal  flight  segments.  Both  of  the  inflight  emergencies  were 
associated  with  an  approximately  50%  increase  in  heart  rate 
about  one  minute  following  the  event  in  both  pilot's,  even 
though  their  flying  base  rates  were  different,  (86  bpm  and  70 
bpm)  .  After  reaching  the  peak,  level,  the  heart  rates  of  both 
pilots  returned  to  pre-emergency  levels  within  60  to  90 
seconds.  The  short  recovery  time  following  these  major 
incidents  reflects  the  high  quality  of  the  r'-iil  and  training 
of  the  pilots.  The  data  from  one  pilot  are  depicted  in  Figure 
2.  This  type  of  event  cannot  be  planned,  and  we  were  fortunate 
that  we  were  collecting  data  when  it  happened.  In  contrast 
to  the  flight  emergencies,  the  simulator  emergencies,  both 
"crashes,"  produced  no  increase  in  heart  rate.  Consequently, 
since  the  simulator  emergency  produced  no  effects,  it  seems 
that  the  only  way  to  gather  this  type  of  data  is  by  being  there 
when  the  event  occurs  (Wilson,  Skelly  &  Purvis,  1988). 
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Figux'e  2,  Csrdisc  vssponsss  to  inflight  Gmergency  ~  bird 
strike.  The  upper  graph  shows  the  mean  heart  rate  for  10 
second  epochs.  The  arrow  designates  the  time  of  the  bird 
strike.  The  lower  graph  depicts  the  interbeat  intervals  during 
the  episode.  Smaller  interbeat  intervals  reflect  increased 
heart  rate.  Note,  in  the  lower  graph,  that  there  is  more 
variability  in  the  interbeat  intervals  before  and  after  the 
strike  and  very  regular  interbeat  intervals  during  and 
immediately  after  the  .strike. 

These  results  dernonstrete  that  physiological  data  can  be 
used  to  measure  the  effects  of  varying  workload  levels  found 
during  flight  and  that  the  effects  of  the  pilot's  position  in  a 
flight  formation  has  effects  upon  his  physiological  data.  The 
emergency  data  give  us  insight  into  the  changes  that  occur  and 
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their  time  course  when  real  emergencies  happen  and  are 
resolved.  These  data  also  provide  a  contrast  between  actual 
flight  and  simulated  flight.  While  the  circumstances  of  the 
simulation  were  not  ideal,  it  is  clear  that  simulation  docS  not 
have  the  same  effects  as  actual  flight.  Different  simulations 
or  a  different  simulation  environment  might  produce  more  varied 
results  but  these  data  do  show  us  that  simulation  and  flight 
effects  can  be  quite  different  and  that  these  effects  can  be 
measured  with  physiological  data. 

B-1  Data 

Data  collected  during  B-1  flights  represent  the  first  time 
that  physiological  data  have  been  collected  from  bomber  crew 
members,  or  any  crew  members,  during  Green  Flag  exercises. 

This  is  a  very  realistic  exercise  which  simulates  actual  combat 
with  multiple  aircraft,  which  represent  both  friendly  and 
hostile  forces.  Not  only  was  this  the  first  time  the  B-1 
participated  at  Green  Flag,  but  the  test  environment  was 
unique.  These  data  showed  the  effects  of  "combat"  on  aircrews 
in  a  new  and  realistic  situation.  The  heart  rate  and  eyeblink. 
data  from  this  study  provided  evidence  concerning  crew  members' 
reactions  to  the  stress  of  and  anticipation  of  "simulated  war" 
as  well  as  their  reactions  to  the  workload  of  the  different 
mission  segments.  This  is  an  interesting  situation  since  only 
training  missions  are  usually  available  for  data  collection  and 
one  must  postulate  what  the  effects  of  the  "real"  mission  will 
be.  Figure  3  shows  mean  heart  rate  from  five  crew  members 
during  their  mission  of  flying  to  the  bombing  range,  entering 
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the  range,  then  orbiting  outside  the  range,  re-entering  the 
range  and  finally  returning  to  base.  The  effects  of  the 
bombing  range  segments  and  the  anticipation  of  entering  the 
range  are  noteworthy  in  this  study.  These  data  represent  the 
first  Green  Flag  mission  for  all  five  crew  members  and  the 
anticipation  of  entering  the  target  area  for  the  first  time  is 

1  10 

100 
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Figure  3.  Mean  heart  rates  for  five  B-1  crew  members 
recorded  during  Green  Flag  exercises .  Note  the  steady  increase 
in  heart  rate  beginning  with  the  enroute  segment  and  peaking  at 
the  bombing  range  when  over  the  target  and  decreasing  with  the 
last  two  range  segments .  The  second  bombing  range  segment  did 
not  show  the  same  anticipation .  The  key  for  the  mission 
segments  is:  Pre  -  pretakeoff  check  lists,  T/0  -  takeoff,  I/R 
-  enroute  to  the  range,  R1  to  R8  -  segments  in  the  bombing 
range,  TARl  and  TAR2  -  over  the  target,  ORB  -  orbiting  between 
runs  on  the  range,  RTN  -  return  flight  to  base,  LAND  -  landing. 
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seen  in  the  steady  increase  in  heart  rate  as  they  approached 
the  range.  This  effect  is  reduced  after  the  first  range  run  and 
the  initial  heart  rates  prior  to  the  second  bombing  run  are 
lower.  The  range  segments  themselves  were  associated  with  high 
heart  rates. 

F-4  Data 

The  F-4  project  was  designed  to  study  the  effects  of  air- 
to-ground  training  missions  on  both  of  the  F-4  crew  members. 
Since  the  F-4  crew  consists  of  a  pilot  and  a  weapons  systems 
officer  (WSO) ,  we  could  study  the  effects  of  being  in  the  same 
place  at  the  same  time  on  two  crew  members  with  different 
duties.  Their  workload  varies  separately  as  a  function  of 
mission  segment  due  to  their  different  jobs.  Although  they 
work  together,  the  two  crew  members  have  different 
responsibilities  with  different  demands.  We  also  collected 
data  while  the  crews  performed  a  laboratory  tracking  task  prior 
to  the  flight.  This  task  has  previously  been  used  in  the 
laboratory  to  study  workload  because  it  has  some  of  the 
attributes  of  flying.  With  this  data,  we  can  compare  each  crew 
members'  laboratory  data  with  their  flying  data  and  gain 
insight  into  just  how  much  the  laboratory  data  can  tell  us 
about  their  responses  during  actual  flight.  This  will  be  very 
helpful  in  evaluating  laboratory  data  for  its  usefulness  to 
flight  situations.  As  can  be  seen  in  Figure  4,  there  were 
large  effects  on  heart  rate  for  both  the  pilots  and  WSOs  due  to 
the  different  flight  segments.  The  range  produced  the  largest 
changes  with  a  mean  increase  of  31  beats  per  minute  (bpm) , 
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which  is  a  45%  increase  over  the  baseline  level  for  the  pilots. 
This  cnange  was  no  doubt  due  to  the  increased  cognitive 
workload,  the  increased  physical  activity  and  the  increased  G- 
forces.  All  of  the  flight  segments  did  not  cause  the  same 
level  of  change  in  the  heart  rates.  In  fact,  the  heart  rate 
changes  correlated  very  well  with  subjective  estimates  of 
mental  workload  for  the  flight  segments.  The  increases  in  the 
pilots'  heart  rates  were  larger  than  those  of  the  WSOs, 
illustrating  the  higher  demands  of  piloting  the  aircraft.  The 
one  noteworthy  exception  was  during  the  segment  in  which  the 
WSO  was  actually  flying  the  aircraft  (CE)  which  produced  a 
larger  increase  in  the  WSO  heart  rates  than  that  of  the  pilots. 
This  effect  is  similar  to  that  reported  with  pilots  and 
copilots  in  multicrew  aircraft  when  the  pilot  in  control  of  the 
aircraft  shows  a  higher  heart  rate  than  when  the  other  pilot  is 
in  control. 

The  laboratory  task  vs  flight  comparisons  were  very 
informative.  The  main  difference  was  the  magnitude  of  the 
changes  in  the  physiological  data.  The  pilots'  heart  rates 
increased  an  average  of  3  bpm  (5%)  during  the  tracking  task 
compared  to  the  baseline  condition,  but  increased  an  average  of 
almost  11  bpm  (16%)  while  going  over  the  preflight  check  lists. 
Takeoff  was  associated  with  a  20  bpm  (30%)  increase  and  the 
mean  during  the  range  segments  approached  100  BPM  (45%) .  The 
magnitude  of  the  increase  was  much  larger  during  the  flight 
segments  than  during  the  tracking  task  suggesting  that  the 
cardiac  dynamics  were  quite  different. 
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Figure  4.  Mean  heart  rates  for  the  10  F-4  pilots  and  10  F-4 
ViSOs.  During  the  air-to-ground  training  flights,  the  pilots' 
heart  rates  were  higher  than  the  WSOs,  even  though  their 
baseline  heart  rates  were  approximately  the  same.  The  one 
exception  during  flight  is  segment  CE  when  the  WSOs  were  flying 
the  aircraft .  Note  that  the  pilot  and  WSO  heart  rates  were 
essentially  identical  during  the  ground-based  tracking  task. 

Two  minutes  of  data  were  averaged  together  for  each  data  point. 
The  segment  abbreviations  are:  EP  -  pilot  only,  evoked 
potential  baseline,  BL  -  resting  baseline,  LI  and  L2  -  low 
difficulty  level  tracking  task.  Ml  and  M2  -  medium  difficulty 
level  tracking  task,  BR  -  pref light  briefing,  PL  -  pre flight 
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check  lists,  PT  -  pretakeoff  prior  to  aircraft  roll,  TO  - 
takeoff,  LE  &  LL  -  low  level  flight,  BT  -  G  maneuver,  R1  to  R4 
-  bombing  range,  JU  -  join-up  after  range,  CE  -  cruise  back  to 
base  WSO  flying,  CC  -  cruise  back  to  base  pilot  flying,  AP  - 
approach  to  landing,  TD  -  touchdown,  TP  -  following  touchdown, 
CL  -  post  flight  check  lists. 

The  eye  blink  data  were  also  very  different  between  the 
laboratory  and  flight  environments.  The  laboratory  tracking 
task,  which  is  a  single,  visual  task,  actually  produced  an 
inhibition  in  the  blink  rate,  as  shown  in  Figure  5.  The  blink 
rate  is  normally  in  the  vicinity  of  20  to  25  blinks  per  minute. 
During  the  tracking  task,  the  mean  blink  rate  for  the  WSOs  was 
less  than  five  blinks  per  minute.  One  WSO  blinked  only  one 
time  during  the  two  minute  tracking  session.  This  means  that 
the  laboratory  data  provided  very  little  information  about  the 
cardiac  or  eye  blink  changes  that  took  place  during  flight. 

More  importantly,  this  means  that  in  order  to  understand  the 
data  collected  from  ambulatory  subjects  while  they  perform 
their  daily  tasks,  we  must  have  a  database  constructed  from 
these  environments. 

We  also  have  collected  the  first  evoked  brain  potentials 
recording  during  actual  flight.  There  are  literally  hundreds 
of  laboratory  evoked  potential  papers  and  the  use  of  evoked 
potentials  to  evaluate  pilot  workload  has  been  discussed  by 
many  investigators  for  a  number  of  years.  While  our  main  goal 
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Figure  5.  Pilot  and  WSO  blink  rates  for  the  F-4  air-to- 
ground  training  missions .  Note  the  large  suppression  of  blinks 
during  the  ground  based,  single  task  tracking  segments  when 
compared  to  the  ground-based  pref light  briefing  and  flight 
data.  The  data  are  from  the  same  crew  members  in  Figure  4  and 
the  segment  legend  is  also  identical . 

was  to  demonstrate  that  evoked  potentials  could  be  recorded 
from  pilots  while  flying,  we  were  pleased  to  find  that  the 
evoked  potentials  provided  meaningful  data  on  the  workload 
experienced  during  the  two  flight  segments.  We  recorded 
auditory  evoked  potentials  while  the  pilots  were  on  the  ground 


14 


during  a  resting  period  and  during  the  tracking  task  prior  to 
flight  to  provide  ground-based  data  for  comparison  with  the 
flight  evoked  potentials.  The  flight  evoked  potentials  were 
collected  from  the  pilots  during  two  cruise  segments,  one  when 
the  pilot  was  flying  the  aircraft  and  one  when  the  WSO  was 
flying  the  aircraft.  The  pilots'  workload  was  higher  when  he 
was  flying  the  aircraft  as  was  supported  by  the  subjective 
workload  ratings.  The  evoked  potentials  from  the  five  segments 
are  shown  in  Figure  6.  These  data  represent  the  grand  mean 
averaged  evoked  potentials  from  the  seven  of  ten  pilots  whose 
data  were  acceptable.  The  evoked  potentials  were  recorded 
between  midline  scalp  parietal  and  central  sites.  The  mean 
amplitudes  of  the  positive  going  component  with  a  latency  of 
about  200  msec  (P2)  is  shown  in  Figure  7.  The  amplitude  of 
this  component  decreased  during  the  highest  workload  segment, 
pilot  flying,  and  not  during  the  other  flight  segment  when  the 
pilot  was  not  flying  the  aircraft.  These  data  demonstrate  that 
with  care  brain  evoked  potentials  can  be  recorded  during  flight 
and  that  they  produce  meaningful  data  with  regarding  the 
pilots'  cognitive  workload.  These  data  encourage  us  to 
continue  collecting  evoked  potentials  during  flight  and  to 
expand  to  other  flight  situations  (Wilson  &  Fullenkamp,  1990; 
Wilson  Fullenkamp  &  Davis,  in  press,  Wilson,  in  press) . 

Flight  Segment  Classification 

While  most  laboratories  record  only  one  type  of 
physiological  data,  heart  rate,  EEG,  or  eyeblink,  for  example, 


15 


Figure  6.  Grand  mean  evoked  potentials  from  seven  pilots 
in  response  to  auditory  tones.  The  evoked  potential  components 
are  labeled  on  the  top  most  average.  The  evoked  potentials 
were  recorded  during  the  BL  -  baseline,  TL  -  low  level 
tracking,  TM  -  medium  level  difficulty  tracking,  LE  -  low  level 
outbound  flight  (pilot  flying)  and  CE  -  cruise  inbound  (WSO 
flying)  . 
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Figure  7.  This  graph  represents  the  mean  amplitude  of  the 
P2  component  of  the  evoked  responses  from  each  of  the  five 
segments .  Note  that  the  only  significant  reduction  in  P2 
amplitude  was  during  the  segment  in  which  the  pilot  was  flying 
the  aircraft  and  not  during  the  segment  when  the  WSO  was 
flying,  nor  during  the  tracking  task  segments . 

we  routinely  collect  several  types  of  physiological  data 
(including  performance  and  subjective  data) .  We  analyzed  these 
data  separately,  and  then  combine  them  to  permit  synthesis  of 
the  several  results.  This  is  illustrated  in  the  F-4  study  in 
which  five  different  physiological  signals  were  collected.  We 
have  tested  methods  of  synthesizing  these  data  to  be  able  to 
gain  additional  information  to  that  available  from  any  one  of 
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the  separate  results.  We  feel  that  this  approach  is  valid 
since  flying  modern  aircraft  is  not  a  single  task  but  places 
demands  on  many  aspects  of  the  human  cognitive  system. 
Consequently,  no  one  physiological  measure  will  be  adequate  to 
characterize  the  mental  workload  of  all  aspects  of  flight. 
Further,  the  physiological  systems  of  individual  pilots  respond 
differently  to  the  workload  and  stresses  of  flying,  thus  by 
making  many  measures  we  are  able  to  accommodate  the  unique 
response  characteristics  of  each  pilot. 

A  recent  example  of  this  approach  was  our  demonstration  of 
the  capability  of  classifying  eight  flight  segments  on  the 
basis  of  the  physiological  data  for  the  F-4  pilots  and  WSOs 
described  above.  Seven  variables  derived  from  their  heart  and 
eyeblink  data  were  submitted  to  stepwise  linear  discriminate 
analysis  in  order  to  determine  the  capability  of  correctly 
classifying  eight  of  the  flight  segments.  Using  the  flight 
data  for  both  the  training  and  test  data  sets  we  were  able  to 
correctly  classify  all  eight  flight  segments  (100%)  for  seven 
of  the  pilots,  90%  of  the  time  for  one  pilot  and  80%  for  the 
other  two  pilots.  The  classification  success  for  the  nine  WSOs 
with  complete  data  was  100%  for  seven  of  them  and  90%  for  the 
other  two.  This  is  a  very  high  rate  of  success  but  it  must  be 
noted  that  the  same  data  set  was  used  for  training  and  testing 
the  classifier  since  we  did  not  have  a  replication  of  the 
mission  segments.  The  jackknife  procedure  was  used  to  provide 
a  conservative  estimate  of  the  goodness  of  classification  and 
was  in  the  neighborhood  of  70%  correct  for  both  pilots  and  WSOs 
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(Wilson  &  Fisher,  1991) .  We  plan  to  collect  two  sets  of  data 
in  our  next  flight  study  so  that  we  will  be  able  to  test  the 
goodness  of  classification  accuracy  using  a  different  test  set. 

These  three  airborne  studies  provide  us  with  a  data  base 
that  is  unique  in  the  number  of  aircraft  types  used  and  the 
roles  of  the  subjects.  Single-seat,  low-level  attack  aircraft, 
two  seat  fighter-bombers  and  multicrew  bombers  were  used.  Many 
of  the  flight  segments  are  similar  for  all  of  these  aircraft; 
takeoff,  cruise,  and  landing,  for  example.  From  this  data 
base,  we  can  identify  similarities  and  differences  across  the 
several  missions.  This  information  will  help  us  generate  a 
picture  of  the  general  effects  of  workload  and  also  the  effects 
of  specific  factors  related  to  mission  type  and  aircraft  type. 
It  is  unusual  that  one  laboratory  has  such  a  diverse  data  base 
and  the  potential  for  enlarging  upon  it.  We  plan  to  continue 
our  flight  studies  in  order  to  enlarge  our  database  and 
increase  the  number  of  unique  situations.  It  is  necessary  to 
collect  inflight  data  since  laboratory  data  has  limited 
usefulness  when  one  wants  to  extrapolate  to  the  realm  of 
flight,  as  was  demonstrated  in  our  F-4  project. 

C-17  Test  and  Evaluation 

We  are  currently  working  with  the  C-17  Test  and  Evaluation 
(T  &  E)  test  team  in  their  evaluation  of  the  C-17.  They  plan 
to  use  heart  rate  and  possibly  eyeblink  measures  as  part  of  the 
operational  T  &  E  plan  for  the  C-17.  This  is  the  first  time 
that  physiological  data  have  been  used  as  an  integral  part  of  a 
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T  &  E  test  plan  to  directly  measure  crew  workload.  Cognitive 
work  overload  is  a  concern  with  the  C-17  since  only  two  crew 
members  will  fly  the  aircraft  rather  than  the  larger  crew  used 
on  existing  transport  aircraft.  Since  no  absolute  overload 
criteria  or  "red  line"  values  exist  for  any  workload  measure, 
our  approach  is  to  use  values  from  similar  mission  events  in 
current  aircraft  to  provide  baselines  to  compare  the  new 
aircraft.  If  the  physiological  responses  are  not  different 
from  the  baseline  aircraft  values  for  the  same  maneuvers,  then 
the  workload  in  the  new  aircraft  is  acceptable.  However,  if 
the  physiological  data  in  the  test  aircraft  deviate 
significantly  from  the  baseline  aircraft,  then  the  workload  can 
be  judged  as  too  high.  We  are  collaborating  with  the  C-17  T  & 
E  team  to  collect  physiological  data  from  their  pilots  during 
proficiency  missions  in  their  currently  certified  C-130  and  C- 
141  aircraft.  Since  these  are  accepted  aircraft,  we  will  be 
able  to  compare  the  physiological  responses  from  their  pilots 
during  the  same  mission  events  in  the  C-17  with  those  in  the 
older  aircraft.  This  will  enable  them  to  determine  if  the 
workload  in  the  C-17  does  or  does  not  exceed  that  of  the  older 
aircraft  during  the  same  mission  events,  e.g.,  takeoff,  air 
drops,  etc.  An  example  of  the  baseline  aata  during  a  high 
workload  Low  Altiltude  Parachute  Extraction  drop  is  shown  in 
Figure  8.  The  eyeblink  record  from  the  pilot  show  the  pattern 
of  blinks  and  their  durations  before,  during  and  after  the  low 
level  drop.  The  inhibition  of  blinks  at  the  drop  show  the 
responses  to  the  extremely  high  visual  workload  when  flying 
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Figure  8.  Cardiac  and  eye  blink  responses  recorded  during 
a  C-130  low  level  cargo  drop.  The  inhibition  of  blinks  and 
decreased  interbeat  intervals  occurred  during  the  drop  and 
provide  measures  of  the  increased  workload  during  this  segment. 
The  vertical  axes  are  IBI  (interbeat  intervals)  for  the  cardiac 
data  and  duration  of  the  eye  blink  closure  for  the  eye  blinks, 
both  in  milliseconds .  The  figure  shows  the  pattern  of 
responses  for  both  measures . 

extremely  close  to  the  ground  and  experiencing  the  radical 
changes  in  aircraft  flying  characteristics  as  the  large  load  is 
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pulled  from  the  aircraft.  fhe  heart  rate  increases  during  this 
maneuver  as  is  shown  by  the  decreased  interbeat  intervals  in 
the  top  panel  of  the  figure. 

Inflight  Voice/Time  Code  Recorder 

During  the  A-7  study,  it  became  obvious  to  us  that  we  needed 
a  better  method  to  mark  and  later  locate  the  time  of  important 
mission  events.  The  times  were  necessary  so  that  we  could 
correlate  the  physiological  data  with  specific  mission  events. 

Our  first  approach  of  tape  recording  aircraft  radio 
communication  was  unsatisfactory;  the  lack  of  a  continuous  time 
record  made  it  difficult  and  time  consuming  to  accurately 
correlate  mission  events  with  the  time  that  the  event  occurred. 
Consequently,  we  developed  a  device  that  simultaneously  records 
aircraft  communications  and  a  time  code.  A  small,  personal, 
commercially  available  stereo  audio  recorder  and  playback  unit 
is  now  used  in  our  flight  studies.  The  aircraft  communications 
are  recorded  on  one  channel  and  a  time  code  from  a 
microprocessor  is  recorded  on  the  other  channel.  The  unit  is 
small  and  can  be  worn  by  a  crew  member  or  placed  in  the 
aircraft.  After  the  mission,  the  taped  signals  are  played  back 
and  a  LCD  display  shows  the  time  as  the  audio  record  is  heard 
so  that  the  time  of  important  mission  events  can  be  determined 
to  an  accuracy  of  one  second.  Several  other  research  groups 
with  similar  requirements  have  expressed  interest  in  this 
device  and  a  patent  application  has  been  submitted,  by  the  Air 
Force,  to  the  U.S.  Patent  Office  (Wilson,  Hall  &  Benadam, 
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1991)  . 


Related  Efforts 

In  order  to  expand  our  range  of  environments  for  data 
collection,  we  have  installed  a  Neuropsychological  Workload  Test 
Battery  at  the  ASD/ENECH  simulator  facility.  We  have  trained 
their  personnel  on  the  use  of  the  system  and  have  collaborated 
on  one  study  evaluating  a  new  display.  The  results  showed  that 
eyeblinks  were  sensitive  to  the  workload  differences  between 
displays  when  performance,  subjective  or  heart  rate  data  were 
not  (Wilson,  Hughes  &  Hassoun,  1990) .  This  cooperative  effort 
afforded  us  access  to  a  simulator  facility  and  will  permit  us 
to  explore  the  application  of  our  measures  in  this  environment. 
Additionally,  it  will  provide  exposure  of  psychophysiological 
methods  to  a  wider  audience  since  ASD/ENECH  works  directly  with 
several  SPOs . 

In  order  to  enhance  information  exchange  between 
researchers  who  collect  psychophysiological  data  in  "real 
world"  environments  and  to  increase  our  own  understanding, 
funding  was  obtained  from  the  European  Community  to  hold  a 
workshop  entitled  "Psychophysiological  Measures  in  Transport 
Operations."  The  meeting  was  held  in  Cologne,  Germany, 

November,  1990.  Nineteen  scientists  presented  their  work  on 
ambulatory  recording  during  different  types  of  transport 
operations.  These  included  airplanes,  buses,  cars,  trucks  and 
locomotives.  Most  of  the  attendees  were  from  Europe  and  three 
were  from  the  United  States.  A  wide  range  of  problem  areas, 
data  collection  and  analysis  methods  were  represented. 
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Selected  papers  from  the  workshop  will  appear  in  a  special 
issue  of  the  journal  Ergonomics  (Wilson,  in  press) . 

LABORATORY 

Our  laboratory  efforts  are  aimed  at  improving  our 
understanding  of  cognitive  workload  and  subject  state  and  also 
to  develop  and  test  new  procedures  to  aid  in  the  measurement  of 
operator  workload  and  state.  These  efforts  include  developing 
techniques  that  will  permit  us  to  better  understand  human 
information  processing  and  to  evaluate  available  procedures  for 
possible  inclusion  in  our  test  battery. 

Heart  Rate  Variability 

An  example  of  our  evaluation  of  existing  procedures  are 
the  comparative  studies  that  we  have  recently  undertaken  to 
evaluate  the  value  of  heart  rate  variability  (HRV) .  HRV  refers 
to  the  variation  in  the  interval  between  heart  beats,  see  the 
beginning  and  ending  portions  of  the  bottom  of  Figure  2  for  an 
example.  This  variability  between  beats  is  thought  to  decrease 
under  conditions  of  higher  cognitive  activity  and  to  increase 
as  the  cognitive  activity  decreases.  There  is  a  large 
literature  based  on  laboratory  results  of  heart  rate 
variability  as  a  measure  of  higher  nervous  system  involvement 
in  various  tasks.  There  are  numerous  methods  used  to  calculate 
HRV;  in  1973  Opmeer  found  26  different  methods  (Opmeer,  1973) . 
Some  of  these  methods  are  commercially  available.  We  have 
tested  four  of  these  methods  using  existing  laboratory  and 
flight  data.  Our  first  question  was:  does  HRV  tell  us  anything 
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more  than  we  can  learn  from  heart  rate  alone?  The  second 
question  was:  if  HRV  is  a  valuable  tool,  which  of  the  methods 
of  calculation  is  best,  or  is  one  better  in  certain  situations 
while  others  have  advantages  in  other  situations?  We  used 
heart  rate  data  collected  in  laboratory  experiments,  data  from 
simulators,  and  data  from  flight  studies.  This  wide  range  of 
data  permitted  us  to  make  definitive  statements  about  the 
utility  of  HRV  and  about. which  of  the  various  methods  is  best 
in  a  given  situation.  To  utilize  the  expertise  available  in 
this  area,  we  held  a  Heart  Rate/Respiration  Workshop  in  Dayton 
during  the  Summer  of  1989  in  which  seven  experts  from  Europe 
and  the  United  States  participated.  The  goal  was  to  have  them 
address  the  issue  of  the  proper  use  of  these  measures  in  Air 
Force  and  other  applied  environments.  The  proceedings  of  this 
workshop  will  appear  as  a  special  issue  of  the  journal 
Biological  Psychology  (Wilson,  in  press) .  We  learned  a  great 
deal  from  the  workshop  concerning  the  collection,  reduction  and 
analysis  of  heart  rate  data  and  by  publishing  the  papers  this 
information  will  be  available  to  others. 

The  results  of  our  testing  the  utility  of  HRV  are  that  it 
does  not  seem  to  add  appreciably  to  conclusions  drawn  on  the 
basis  of  simple  measures  of  heart  rate.  While  heart  rate  data 
displayed  graded  differences  between  the  several  levels  of 
workload  in  flight,  HRV  measures  seem  to  be  dichotomous.  For 
example,  the  HRV  shown  in  Figure  9  contrasts  with  the  heart 
rate  data  from  the  same  flight  segments  shown  in  Figure  4. 

While  the  heart  rate  data  show  several  levels  corresponding  to 
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the  workload  levels  of  the  ground  and  flight  segments,  the  HRV 
data  show  only  two  or  possibly  three  levels.  Also,  from  the 
variability  data,  it  is  not  clear  why  the  tracking  task  is 
associated  with  the  same  degree  of  reduction  of  HRV  as  the 
bombing  range  segments  during  flight-  It  appears  that  the 
variability  measure  is  indicative  of  the  presence  of  cognitive 
activity,  but  does  not  give  information  about  the  degree  of 
cognitive  involvement.  This  limits  the  application  of  HRV  in 
our  environment  since  we  require  graded  measures  of  mental 
effort  and  not  just  an  indicator  of  its  presence  or  absence 
(Wilson,  1991) . 

Further,  in  a  study  explained  below,  we  assessed  the 
effects  of  learning  a  task  and  adaptation  to  the  laboratory 
while  subjects  performed  a  mathematics  task.  We  found  that 
heart  rate  changes  were  associated  with  learning  the  task, 
adaptation  to  the  laboratory,  and  the  difficulty  levels  of  the 
task,  while  the  HRV  measures  were  sensitive  only  to  the 
adaptation  to  the  laboratory  and  not  task  difficulty  (Wilson  & 
McCloskey,  1986) . 

With  regard  to  the  issue  of  which  of  the  several  measures 
of  heart  rate  variability  to  use,  we  have  concluded  that  those 
we  tested  seem  to  be  equivalent  to  one  another.  Only  small 
differences  were  found  between  the  results  of  these  methods 
that  were  inconsequential.  This  has  been  borne  out  by  a  recent 
article  which  also  compared  several  measures  on  laboratory  data 
(Grossman  &  Wientjes,  1991).  However,  the  choice  of  method  may 
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Figure  9.  Mean  heart  rate  variability  for  the  10  pilots 
during  F-4  air-to-ground  training  missions .  The  data  are 
plotted  relative  to  the  ground-based  resting  segment.  Note 
that  only  two  or  possibly  three  levels  of  HRV  are  shown  in 
contrast  to  the  more  varied  response  levels  of  the  heart  rate 
data  shown  in  Figure  4.  The  segment  legend  is  explained  in 
Figure  4.  Two  bands  of  HRV  are  represented,  0.06  -  0.10  Hz, 
i.e.,  the  blood  pressure  or  Traube-Hering-Mayer  wave,  and  0.12 
-  0.40  Hz,  i.e.,  the  respiration  band.  The  responses  in  these 
two  bands  are  statistically  equivalent . 


be  governed  by  practical  considerations  such  as  the  length  of 
the  available  data  record.  The  spectral  analysis  techniques, 
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by  nature,  require  a  minimum  of  three  to  five  minutes  of  data 
to  sufficiently  resolve  the  lowest  frequency  bands.  In  many 
situations,  especially  in  the  "real  world,"  only  shorter  epochs 
of  data  are  available.  The  methods  that  use  filtering 
techniques  have  the  advantage  in  these  situations  since  they 
are  valid  with  shorter  segments  of  data.  However,  if  one  is 
interested  in  the  relationship  between  cardiac  and  respiratory 
activity,  the  spectral  and  direct  measurement  techniques  have 
the  advantage  since  they  permit  the  direct  comparison  of  the 
cardiac  and  respiration  activity. 

Learning  and  Adaptation  Effects  on  Heart  Rate 

With  regard  to  cardiac  measures  of  workload  in  the 
laboratory,  one  line  of  current  inquiry  arose  from  our 
inability  to  produce  heart  rate  changes  in  our  subjects  as  a 
result  of  cognitive  task  difficulty  manipulations  (McCloskey, 
1987;  Wilson,  et  al.,  1986;  Yolton,  et .  al.,  1987).  We 
concluded  that  there  were  significant  differences  between  our 
experimental  methodology  and  that  of  the  studies  reported  in 
the  literature.  While  the  subjects  in  the  studies  reported  in 
the  literature  were  usually  naive  to  the  task  used,  our 
subjects  were  practiced  to  a  performance  criteria  prior  to 
physi'^logical  data  collection.  In  gaining  experience  with  the 
task,  our  subjects  also  became  adapted  to  the  laboratory 
environment  while  subjects  from  the  other  studies  were  in  the 
laboratory  only  on  the  day  of  testing.  This  led  us  to  perform 
a  study  with  naive  subjects  from  whom  we  recorded  heart  rate 
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and  respiration  as  they  practiced  the  task  to  criterion 
performance  levels.  As  we  had  predicted,  there  were 
significant  task  difficulty  effects  early  in  the  training, 
prior  to  achieving  criteria,  that  went  away  with  practice,  see 
Figure  10  (Wilson  &  McCloskey,  1986) .  These  results  are  very 
important  since  they  show  a  context  effect  which  severely 
restricts  the  application  of  previous  laboratory  results.  It 
suggests  that  typical  laboratory  data  may  have  limited  utility 
in  predicting  "real  world"  effects  since  operators  are 
typically  highly  overtrained  and  quite  adapted  to  their  work 
environment.  In  addition  to  these  findings,  we  have  found  that 
flight  data  is  quite  different  from  laboratory  data  in  terms  of 
the  magnitude  of  change.  The  range  of  heart  rates  reported  in 
laboratory  studies  may  be  6  to  10  bpm  (approximately  10%)  and 
exhibit  statistical  significance.  As  discussed  above,  we 
routinely  find  up  to  40%  to  50%  increases  in  heart  rates  during 
normal  flights!  A  new  body  of  data  will  be  necessary  from  the 
"real  world"  so  that  this  type  of  data  can  be  properly 
interpreted. 

We  have  also  begun  collecting  respiration  data  in  addition 
to  heart  rate.  One  component  of  the  HRV  is  driven  by 
respiration  activity,  0.12  -  0.40  Hz.  There  is  reason  to 
believe  that  the  respiration  signal  itself  should  be  analyzed 
since  it  drives  this  component  of  the  HRV  and  may  be  a  better 
measure  of  workload  effects.  We  have  collected  respiration 
data  in  both  laboratory  and  flight  studies.  Dr  Richard  Backs, 
Wright  State  University,  performed  a  study  in  our  laboratory 


29 


HEART  RATE  (bpm) 


76^ 


74  - 
72  - 
70  - 

68  H 
66  ^ 

64  - 
62  - 

i 

BL1  2  3  4  5  6  7  8  9  101112131415  RP 

TRIAL  BLOCK 

Figure  10.  Heart  rate  measures  from  the  two  groups  of 
subjects  during  15  three-minute  trials  of  learning  a 
mathematics  task  having  two  difficulty  levels  (task  group)  or 
watching  a  video  tape  (video  group) .  The  heart  rates  from  the 
two  groups  decrease  together  over  trials  until  trial  block  8 
when  the  video  group  heart  rates  decrease  more  rapidly.  Within 
the  task  group  only  the  first  four  trial  blocks  exhibited 
significant  differences  between  the  low  and  medium  difficulty 
levels  of  the  task. 

and  is  continuing  this  research  in  his  laboratory  with  funding 
from  the  Air  Force.  He  is  investigating  the  relationship 
between  increases  in  cognitive  and  physical  workload  on  cardiac 
and  respiratory  activity.  Both  of  these  variables  are 
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influenced  by  workload  manipulations  and  the  goal  of  his  work 
is  to  understand  these  relationships  using  single  and  dual  task 
paradigms . 

Investigation  of  Brain  Activity 

Since  the  brain  is  the  organ  responsible  for  cognition, 
direct  measurement  of  the  activity  of  this  organ  during 
cognition  should  lead  us  to  a  better  understanding  of  both 
brain  function  and  cognition.  The  electrical  and  magnetic 
activity  of  the  intact  human  brain  can  be  studied  in  several 
ways.  One  is  to  monitor  the  on-going  activity  by  performing 
spectral  analysis  on  the  EEC  and  searching  for  changes  in  the 
frequency  composition  of  these  signals.  Another  procedure  is 
to  measure  the  small  amplitude  changes  in  brain  activity  that 
are  evoked  by  environmental  stimuli.  Since  these  small 
amplitude  potentials  are  overshadowed  by  the  on-going  brain 
activity,  ensemble  averaging  is  used  to  extract  the  signal  from 
the  background  EEC  or  noise. 

Probe  Evoked  Potentials 

Using  evoked  potential  methods  on  brain  wave  data  we  have 
performed  several  studies  using  the  so-called  "probe"  evoked 
potential  technique.  The  "probe"  stimuli  are  superimposed  upon 
an  ongoing  task  and  the  brain  response  to  these  probe  stimuli 
is  used  as  a  measure  of  involvement  in  the  primary  task.  The 
probe  stimuli  are  non-intrusive  in  that  no  behavioral  response 
or  attention  to  the  probes  is  required  by  the  subject.  The 
assumption  is  that  the  level  of  processing  of  the  task- 
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irrelevant  probe  depends  upon  the  cognitive  resources  remaining 
after  processing  of  the  primary  tasK.  If  the  primary  task  is 
more  demanding  upon  cognitive  resources,  then  the  brain 
response  to  the  probe  will  be  diminished.  Our  results  to  date 
have  shown  that  the  probe  must  be  presented  during  the  actual 
time  of  active  involvement  with  the  primary  task.  That  is,  if 
a  task  is  used  which  has  discrete  trials  in  which  stimuli  are 
presented  periodically  and  the  probe  stimuli  are  randomly 
presented,  small  changes,  if  any,  in  the  probe  evoked  response 
will  be  found.  This  occurs  since  there  is  no  competition  for 
cognitive  resources  when  the  probe  stimuli  occur  during  times 
when  the  subject  is  not  processing  task  information  but  is 
waiting  for  the  next  stimulus.  If  the  occurrence  of  the  probe 
stimuli  is  synchronized  to  the  occurrence  of  the  task  stimuli, 
e.g.,  250  milliseconds  following  task  stimulus  onset,  then  the 
evoked  potentials  to  the  probe  stimuli  will  be  reduced  in 
amplitude  (Wilson  &  McCloskey,  1988) .  Figure  11  shows  the 
amplitude  changes  found  in  probe  evoked  potentials.  Prior  to 
our  work  this  point  was  not  clear. 

The  F-4  evoked  potentials,  described  above,  were  recorded 
using  the  probe  paradigm  in  which  tones  were  presented  while 
the  pilots  were  engaged  in  the  tracking  task,  flying  the 
aircraft  or  while  the  WSO  flew  the  aircraft.  The  probe  evoked 
potential  P2  component  amplitudes  were  reduced  during  the  pilot 
flying  segment  and  demonstrates  that  if  continuous  processing 
tasks  are  used,  then  the  probe  technique  can  be  used  to  monitor 
operator  workload.  Theoretically,  the  probe  paradigm  could  be 
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Figure  11.  Evoked  potentials  to  probe  stimuli  that  were 
presented  250  msec  ifter  the  onset  of  the  primary  task  stimuli 
in  a  mathematics  task.  The  baseline  evoked  potential  was 
collected  during  a  probe-only  condition .  The  reduction  of  the 
amplitude  of  the  early  components  can  be  seen  in  the  figure. 

used  to  define  the  temporal  limits  of  cognitive  processing  of 
the  primary  task.  We  plan  to  continue  to  use  this  method  in 
"real  life"  situations  to  test  its  utility  in  other  situations. 

Task  Stimuli  Evoked  Potentials 

Another  approach  to  using  evoked  brain  responses  to 
measure  cognitive  activity  is  to  record  the  evoked  potentials 
to  the  task  stimuli.  The  probe  stimuli  are  superimposed  on  an 
ongoing  task,  in  this  case  the  evoking  stimuli  are  the  actual 
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task  stimuli  that  the  subjects  must  process.  We  have  used  this 
approach  to  investigate  how  the  brain  processes  information  in 
situations  with  different  difficulty  levels.  A  number  of 
cognitive  tasks  have  been  used,  most  of  them  from  the  Criterion 
Task  Set  (CTS)  developed  by  Shingledecker  in  our  Branch  (1984) . 
A  consistent  result  from  these  studies  is  that  the  late  evoked 
potential  components  are  reduced  in  amplitude  as  the  task 
difficulty  increases.  An  example  of  this  is  seen  in  Figure  12 
which  shows  evoked  potentials  from  a  midline  parietal  electrode 
(Pz)  from  two  levels  of  difficulty  of  a  linguistic  task.  The 
literature  on  this  topic  is  small  and  contradictory.  We  are 
not  certain,  yet,  why  we  find  amplitude  reductions  in  the  task 
stimuli  evoked  potentials  but  we  have  several  hypotheses  that 
we  plan  to  test.  For  example,  it  could  be  that  the  increased 
task  difficulty  results  in  more  distributed  cortical  activity 
that  results  in  a  reduction  at  any  one  electrode  site.  Another 
explanation  is  that  the  latencies  of  the  individual  brain 
responses  to  the  stimuli  are  more  variable,  as  is  sometimes 
found  in  performance  measures  as  increased  variance  in  the 
reaction  times.  This  would  tend  to  reduce  the  component 
amplitudes  as  a  result  of  the  averaging  process.  We  plan  to 
test  these  hypotheses  in  future  experiments. 

Electrical  Brain  Mapping 

Typically,  evoked  potential  studies  use  a  small  number  of 
scalp  electrodes,  usually  three  to  six.  These  data  provide  a 
time  history  of  the  electrical  potential  changes  following 
stimulus  presentation  while  the  brain  is  processing  the 
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Figure  12.  Averaged  evoked  potentials  from  the  parietal 
midline  electrode  (Pz)  recorded  during  the  performance  of  a 
linguistic  task  having  two  levels  of  difficulty .  Note  the 
reduced  amplitude  of  the  large,  late  positive  component  in  the 
more  difficult  level  of  the  task.  The  averages  represent  1024 
msec  of  activity  and  the  vertical  line  is  at  stimulus  onset. 

information.  Current  software  and  hardware  technology  make  it 
possible  to  simultaneously  record  from  many  more  scalp  sites. 
Commercially  available  systems  permit  the  recording  of  20  to  32 
channels  and  some  laboratories  have  systems  with  up  to  128 
channels.  These  mt,thods  add  the  spatial  dimension  to  the  time 
dimension  formerly  available.  With  these  systems,  one  can  see 
the  potential  changes  over  tl.e  head  as  they  change  in  time 
enabling  explorat  on  of  the  functional  neuroanatomy  with  brain 
topographical  techniques.  That  is,  we  are  able  to  watch  the 
time  history  and  see  where  on  the  surface  of  the  scalp  the 
brain  activity  is  taking  place  as  information  is  being 
processed.  We  have  used  our  19-channel  EEC  brain  mapping 


system  in  several  evoked  potential  and  on-going  EEG  studies  to 
better  understand  how  the  brain  processes  information  in 
different  types  of  tasks  and  under  different  levels  of 
difficulty  or  workload. 

The  advantage  of  using  electrical  brain  mapping  was 
demonstrated  in  one  study  in  which  the  subjects  performed  a 
spatial  rotation  task  with  three  levels  of  difficulty.  It  was 
possible  to  follow  the  flow  of  information  processing  over  the 
head  by  following  the  spatial  location  of  the  evoked  potential 
components  over  time.  Reduction  in  electrical  activity  was 
again  associated  with  increased  task  difficulty.  The  utility 
of  the  brain  mapping  procedure  was  shown  when  activity  in  the 
right  hemisphere  central  cortex  was  located  that  would  not  have 
been  seen  from  the  typical  midline  electrode  sites  (see  Figure 
13)  (Wilson,  Swain  &  Davis,  1988) . 

In  another  study,  three  different  tasks,  each  having  two 
levels  of  difficulty,  were  performed  by  the  subjects  while  we 
recorded  19  channels  of  EEG.  This  permitted  us  to  determine 
the  effects  of  different  levels  of  task  difficulty  in  each  of 
three  tasks  so  that  we  could  determine  if  the  evoked  potential 
changes  would  be  universal  across  tasks  or  unique  to  each  task. 
Further,  the  topographic  data  allowed  us  to  follow  the  flow  of 
information  in  the  brain  during  each  task  and  evaluate  how  it 
differed  across  tasks.  The  tasks  were  a  stimulus  degradation 
task,  a  mathematical  processing  task,  and  a  linguistic  task. 

The  effect  of  increased  task  difficulty  was  associated  with 
reductions  in  the  amplitude  of  the  late  evoked  potential 
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Figure  13.  Topographic  map  of  a  late  positive  component 
of  the  evoked  potential  recorded  during  the  processing  of  a 
spatial  orientation  task.  The  maximal  positivity  is  seen  in 
the  right  hemisphere  central  and  parietal  midline  areas.  The 
right  central  positivity  would  have  been  missed  if  the 
recording  sites  were  restricted  to  only  midline  electrodes . 


components  in  all  three  tasks.  The  stimulus  degradation  task, 
which  manipulated  earlier  perceptual  processing,  was  also 
associated  with  larger,  early  components  to  the  degraded 


stimuli.  Apparently,  the  degraded  stimuli  produced  increased 
activity  early  during  the  perceptual  states  of  processing,  but 
were  associated  with  decreased  amplitudes  in  the  later 
components  associated  with  evaluation  and  decision  making.  The 
pattern  of  activity  over  the  head  was  quite  different  among  the 
tasks  and  could  be  used  to  characterize  each  task  (Wilson, 
Palmer,  Oliver  &  Swain,  1991) .  For  example,  the  brain  areas 
associated  with  significant  differences  between  evoked 
potential  components  due  to  differences  in  task  difficulty  as 
shown  in  Figure  14.  These  data  represent  the  pattern  of 
component  differences  due  to  task  difficulty  for  two  tasks, 
stimulus  degradation  and  mathematical  processing.  These 
patterns  show  the  differences  in  the  cortical  areas  activated 
and  in  the  temporal  sequence  of  their  activation.  This  sort  of 
analysis  permits  us  to  begin  developing  an  atlas  of  functional 
brain  topography.  With  this  atlas,  we  will  be  able  to  determine 
the  type  of  task  being  performed  from  the  pattern  of  cortical 
area  activation.  This  will  permit  us  to  determine  similarities 
and  differences  in  cognitive  function  between  different 
cognitive  tasks.  With  this  information,  we  will  be  able  to 
derive  a  much  better  understanding  of  human  cognitive  brain 
functioning . 

Dr  Victoria  Nasman  and  Barbara  Palmer  have  conducted  a 
study  n  which  they  utilized  electrical  brain  mapping  techniques 
to  investigate  the  changes  in  evoked  activity  while  subjects 
performed  a  language  task  with  increasing  levels  of  difficulty. 
The  task  stimuli  were  drawn  from  the  same  set  of  four  letters 
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Figure  14.  Patterns  of  electrical  activity  that  represent 
the  differences  between  the  two  levels  of  task  difficulty  for 
two  tasks.  The  pattern  for  the  stimulus  degradation  task  is  on 
the  left  and  that  for  the  mathematics  task  is  on  the  right. 

for  each  level  of  difficulty  and  the  only  difference  was  the 
subject's  task.  In  each  case,  two  letters  were  presented 
simultaneously.  Subjects  were  asked  to  indicate  by  pressing 
one  of  two  buttons  if  the  letters  were  (1)  physically 
identical,  (2)  same  name  regardless  of  case  and  (3)  both  vowels 
or  consonants.  The  evoked  potential  amplitudes  were  smaller 
for  the  more  difficult  levels  of  this  task.  However,  the 
evoked  response  amplitudes  were  larger  for  the  match  than  the 
not-match  categories  of  the  levels.  The  evoked  potentials 
associated  with  a  simple  reaction  time  task  to  the  same 


stimulus  set  were  notably  lacking  long  latency  components. 

These  later  components  of  the  evoked  response  are  related  to 
the  longer  duration  cognitive  processing  in  the  three  levels  of 
the  language  task  (Nasman,  Palmer  &  Wilson,  1991) . 

The  differences  in  the  pattern  of  evoked  electrical 
activity  between  visual  and  auditory  discrimination  tasks  were 
investigated  by  Dr  Celia  Oliver.  This  study  was  aimed  at  using 
the  N2  response  of  the  evoked  potential  to  study  discrimination 
and  decision  processes  in  humans.  The  N2  cortical  spatial 
locus  is  purportedly  modality  specific,  while  the  location  of 
the  P3  is  the  same  for  all  modalities.  Our  goal  was  to  test 
this  hypotheses  using  brain  mapping  techniques.  The  results 
showed  that  the  location  of  the  early,  perceptual,  components 
was  determined  by  the  modality  of  the  stimulus,  temporal- 
central  for  the  auditory  task  and  occipital  for  the  visual 
task.  With  the  progression  of  time,  the  location  of  maximal 
electrical  activity  in  the  evoked  potentials  moved  toward  the 
midline  parietal  areas  where  the  large  amplitude,  long  latency 
component,  P3,  was  located  for  both  types  of  stimuli.  The  N2 
components  were  located  at  different  sites  due  to  stimulus 
modality  as  was  predicted.  These  data  show  that  the  source  of 
cognitive  brain  activity  early  in  the  processing  sequence  is 
modality-specific  but  common  association  cortex  (parietal) 
areas  are  used  late  in  the  sequence.  This  confirms  concepts  of 
the  functional  anatomy  of  the  brain  and  gives  us  more 
information  concerning  how  the  brain  processes  information  in 
this  type  of  discrimination  task. 
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A  different  type  of  question  was  asked  in  another  brain 
mapping  experiment.  The  question  was:  is  there  electrically 
recordable  activity  in  temporal  and  parietal  areas  of  the  brain 
in  response  to  stimuli  presented  to  the  visual  periphery?  This 
work  has  implications  for  the  understanding  of  the  secondary 
visual  system.  Both  animal  and  human  studies  show  evidence  for 
a  second  visual  system  driven  by  the  periphery  of  the  visual 
field.  This  is  potentially  important  in  the  area  of  display 
design,  for  enhancing  information  processing  capabilities,  and 
because  of  the  interactions  with  the  vestibular  system,  it  may 
provide  a  tool  in  the  study  of  the  perception  of  normal  body 
orientation,  as  well  as  motion  and  space  sickness.  If  we  can 
better  understand  this  system,  then  we  may  be  able  to  use  it  as 
another  channel  for  presenting  visual  information.  This  could 
be  especially  useful  as  we  place  higher  loads  on  the  foveal 
visual  system.  By  understanding  the  peripheral  system,  we  will 
be  able  to  utilize  its  capabilities  to  our  advantage. 

Dr  Gundel,  from  the  Institute  for  Aviation  Medicine, 
Cologne,  Germany,  recently  completed  a  year  with  us.  He 
investigated  the  EEG  cross-power  relationships  over  the  brain 
as  subjects  performed  several  cognitive  tasks.  The  goal  of 
this  project  was  to  find  the  interrelationships  between 
different  areas  of  the  brain  during  active  cognitive  processing 
of  different  tasks  using  on-going  EEG.  Spectral  analysis  was 
performed  on  the  19  channels  of  EEG  data  while  the  subjects 
performed  the  tasks.  The  cross-power  between  all  combinations 
of  leads  for  the  traditional  EEG  frequency  bands  was 
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Figure  15.  Cross-power  maps  for  the  differences  in  alpha 
cross-power  between  the  two  levels  of  difficulty  of  the  visual, 
probability  monitoring  task.  Each  enclosed  figure  represents 
the  cross-power  for  that  electrode  with  the  other  electrodes . 
The  pattern  represents  the  electrode  si.^j  on  the  head  with  the 
nose  at  the  top  and  the  back  of  the  head  at  the  bottom.  The 
length  of  the  negative  signs  represents  the  degree  of  the 
difference  between  the  cross-power  to  the  two  task  levels.  The 
circled  negative  signs  indicate  that  the  magnitude  of  the 
difference  was  statistically  significant.  The  preponderance  of 
the  differences  are  over  the  back  of  the  head. 

calculated.  An  example  of  the  results  for  the  differences  in 
cross-power  between  difficulty  levels  for  the  probability 
monitoring  task  is  shown  in  Figure  15.  The  minus  signs  that 
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are  circled  designate  electrode  pairs  that  showed  significantly 
reduced  alpha  power  in  the  more  difficulty  task  level,  as 
compared  to  the  less  difficulty  level,  over  the  parietal  and 
occipital  electrode  sites.  The  data  showed  that  the  different 
areas  of  the  brain  were  active  during  the  different  tasks  and 
that  the  relationships  between  these  cortical  areas  were  also 
determined  by  the  task  being  performed  (Gundel  &  Wilson,  1991) . 

Task  Classification  Using  EEC  Data 

In  another  approach,  Lt  Col  Frank  Fisher  has  utilized  the 
on-going  EEC  to  classify  the  cognitive  tasks  that  subjects  were 
performing.  This  approach  is  similar  to  the  one  used  for 
classifying  flight  segments  that  was  described  above.  However, 
in  this  case,  we  used  only  EEC  data  from  19  leads.  Spectral 
analysis  was  performed  on  these  data  and  the  magnitudes  of  the 
63  integer  frequencies  between  0  and  63  Hz  were  submitted  to  a 
stepwise  linear  discriminant  analysis.  This  has  not  been  done 
before  because  of  the  very  large  data  set  required,  63 
frequency  values,  19  leads  and  14  task  conditions.  The  EEC  data 
were  grouped  in  several  ways,  each  of  the  63  frequency 
magnitude  values,  the  five  standard  EEC  bands,  seven  EEC  bands 
(the  standard  five  plus  two  other  frequency  bands  above  the 
highest  standard  band)  and  various  groupings  of  the  63  Hz  data, 
e.g.,  grouping  two  adjacent  frequencies,  three  adjacent 
frequencies  up  to  12  adjacent  frequencies.  Each  60  second  data 
set  was  divided  into  two  smaller  sets,  training  and  learning 
sets,  by  assigning  every  other  10  second  epoch  to  one  group. 
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The  highest  correct  classification  of  the  test  data  was  found 
when  using  the  seven  bands,  the  five  traditional  bands  plus  the 
two  higher  bands.  The  percentage  of  correctly  classified 
cognitive  tasks  was  69%  for  the  seven  band,  57%  for  the 
traditional  bands,  41%  for  the  63  Hz  integer  bands  and  the 
highest  percent  correct  for  the  adjacent  frequency  groupings 
was  42%  (Wilson  &  Fisher,  1991). 

Recent  advances  in  EEG  analysis  technology,  partially 
spurred  by  advances  in  MEG,  make  it  possible  to  estimate  the 
cortical  source  of  particular  aspects  of  the  EEG  and  evoked 
potentials.  These  techniques  are  more  precise  than  mapping 
surface  potential  field  maps  shown  earlier  in  Figure  13,  even 
with  the  Laplacian  correction.  Using  these  techniques,  we  plan 
to  be  able  to  better  determine  the  cortical  sources  of  activity 
during  the  processing  of  different  tasks  and  to  gain 
understanding  of  how  increased  workload  in  the  form  of 
increased  task  difficulty  effects  the  sources  of  this  activity. 
We  have  acquired  the  software  package  developed  by  Dr  Michael 
Scherg  which  interfaces  with  our  brain  mapping  system  so  that 
we  can  readily  perform  the  source  analysis  on  the  topographic 
data . 

Using  Combined  Physiological,  Performance,  and  Subjective 
Measures 

We  have  undertaken  studies  to  explore  the  relationship 
between  physiological,  performance  and  subjective  measures 
recorded  while  subjects  perform  CTS  tasks  (see  Figure  16).  The 
several  studies  that  we  conducted  show  discrepancies  between 


the  physiological  and  performance/subjective  data.  The 
physiological  data  often  will  discriminate  between  low  and 
medium  difficulty  levels  but  not  between  medium  and  high. 

Since  the  task  difficulty  levels  were  selected  on  the  basis  of 
reaction  time  and  subjective  estimates  of  difficulty,  it  is  not 
surprising  that  these  two  measures  are  highly  correlated.  The 
lack  of  agreement  of  the  physiological  data  could  have  at  least 
two  explanations.  The  first  is  that  the  physiological  measures 
are  not  sensitive  enough  and  suffer  from  a  ceiling  effect.  It 
is  also  possible  that  the  performance  and  subjective 
differences  are  not  due  to  increased  workload,  but  merely  time. 
That  is,  the  most  difficult  level  of  the  tasks  involve  the  same 
workload  levels,  but  due  to  the  nature  of  the  tasks  these 
activities  are  performed  over  a  longer  period  of  time  producing 
longer  RTs  and  increased  subjective  estimates  of  difficulty. 

One  test  of  these  two  possibilities  would  be  a  dual  task 
situation  in  which  tasks  utilizing  the  same  resources  are 
simultaneously  performed.  One  task  would  remain  constant 
(secondary  task) ,  while  the  workload  of  the  other  task  (primary 
task)  would  be  increased.  Linear  decrease  of  performance  in 
all  levels  of  the  secondary  task  would  support  the 
insensitivity  hypothesis  while  a  leveling  off  of  performance 
decrement  to  the  secondary  task  between  medium  and  high 
difficulty  levels  of  the  primary  task  would  support  the 
physiological  measures. 
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Figure  16.  Task  stimuli  evoked  potentials  during  a 
mathematics  processing  task  having  three  levels  of  difficulty . 
Note  the  decrease  in  amplitude  during  the  medium  and  high 
difficulty  levels  compared  to  the  low  level. 

MAGNETOENCEPHALOGRAPHY 

The  magnetoencephalograph  (MEG)  detects  the  small  magnetic 
fluctuations  associated  with  brain  activity.  This  new 
technology  is  being  applied  to  the  study  of  brain  functioning 
in  several  areas.  Our  MEG  efforts  are  aimed  at  utilizing  the 
spatial  localizing  capabilities  of  the  MEG  to  better  understand 
how  the  brain  processes  information.  This  device  permits  us  to 
examine  the  magnetic  activity  from  a  smaller  area  of  brain 
tissue  than  is  possible  with  standard  electrical  recording 
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techniques.  This  spatial  resolution  not  only  allows  us  to 
better  estimate  the  source  of  brain  activity  but  it  also 
eliminates  some  of  the  "brain  noise"  that  is  seen  in  electrical 
recordings.  This  can  be  seen  in  Figure  17  which  represents 
simultaneous  recordings  of  MEG  and  EEG  evoked  responses  to  the 
same  stimuli  with  increasing  cognitive  demands.  The  magnetic 
and  electrical  evoked  responses  are  essentially  identical  until 
the  cognitive  task  requires  higher  level  decisions  then  added 
components  are  seen  in  the  electrical  recordings  but  not  in  the 
MEG  records.  This  is  probably  the  result  of  the  source  of 
activity  of  the  later  components  being  farther  away  from  the 
MEG  sensor.  Due  to  the  greater  spatial  sensitivity  of  the  MEG, 
the  late  component  is  not  detected  by  the  MEG,  while  the  less 
spatially  sensitive  EEG  detects  both. 

MEG  Evoked  Responses  to  Color 

We  have  shown  that  the  brain  processes  color  information 
in  an  area  of  the  occipital  lobe  that  is  anterior  to  the 
primary  projection  areas.  Further,  for  the  first  time,  it  was 
possible  to  draw  parallels  with  data  from  monkeys  showing 
separate  color  processing  areas  that  have  been  demonstrated  to 
be  crucial  for  the  perception  of  color,  not  just  its  reception. 
Thus,  we  are  also  able  to  provide  additional  support  to  a  new 
theory  of  vision  based  on  animal  data  (Wilson,  Peio  and  Menu, 
1985) . 
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Figure  17.  Simultaneously  recorded  magnetic  evoked  fields 
(solid  line)  and  electrical  evoked  potentials  (dashed  line)  to 
the  same  auditory  tones  while  requiring  five  different  types  of 
cognitive  activity,  from  the  subject.  Due  to  the  differential 
sensitivity  of  the  magnetic  activity,  the  late  positive 
components  associated  with  increased  levels  of  cognitive 
activity  were  not  seen  in  the  magnetic  responses . 
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Figure  18.  Amplitudes  of  the  magnetic  P2M  component  of 
the  auditory  probe  evoked  responses  while  subjects  listened  to 
only  the  probe  stimuli  (control)  or  performed  a  language  task 
(phonological) .  There  was  a  greater  reduction  in  the  amplitude 
of  this  component  in  the  left,  language,  hemisphere  during  the 
performance  of  the  language  task  than  in  the  right  hemisphere. 

MEG  Probe  Evoked  Response  Studies 

We  have  used  the  probe  method  described  above  for  the  EEG 
studies  with  the  MEG.  Our  work  with  probe  evoked  magnetic 
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fields  has  shown  several  things.  The  first  study,  with 
auditory  probes  presented  during  a  language  task,  confirmed 
that  the  differential  hemisphere  reduction  in  the  amplitude  of 
the  probe  response  was  indeed  occurring  in  the  left  (language) 
hemisphere.  As  predicted,  the  evoked  fields  to  the  probe 
stimuli  decreased  in  amplitude  more  over  the  left,  language, 
hemisphere  compared  to  the  changes  in  rhe  right  hemisphere  (see 
Figure  18)  (Papanicoulaou,  et  al,  1988)  . 

In  our  second  MEG  probe  study,  we  found  that  interpreting 
visual  MEG  responses  was  not  as  straightforward  as  it  had  been 
for  the  auditory  data.  In  the  visual  study,  checkerboard  probe 
flashes  were  superimposed  upon  the  stimuli  of  a  spatial 
rotation  task.  Magnetic  and  electrical  evoked  responses  to  the 
probe  flashes  were  simultaneously  recorded  from  over  left  and 
right  occipital  areas  of  the  head.  While  the  electrical 
responses  demonstrated  the  expected  lateral  asymmetrical 
reductions  to  the  probe  in  the  right  hemisphere,  the  MEG 
responses  did  not  show  this  asymmetrical  reduction  (see  Figure 
19).  The  implication  is  that  the  less  spatially  discriminable 
electrical  recordings  were  a  mixture  of  occipital  and  parietal 
cortex  responses  while  the  MEG  was  only  reflecting  occipital 
cortex  activity.  The  reduction  in  electrical  response,  both 
ours  and  those  reported  in  the  literature,  may  be  due  to 
reduced  resource  availability  in  the  parietal  area  which  is 
primarily  responsible  for  information  processing  in  the  spatial 
rotation  task. 
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Figure  19.  (A)  Mean  amplitudes  for  the  electrical  P2 

component  and  (B)  the  magnetic  P2M  component  in  response  to 
visual  probe  stimuli  while  performing  a  spatial  rotation  task. 
The  amplitudes  showed  reductions  from  the  probe  only-condition 
(probe)  to  the  counting  of  the  probe  stimuli  while  the  rotation 
stimuli  were  present  (control)  to  the  actual  performance  of  the 
spatial  rotation  task  (task) .  The  electrical  responses  showed 
a  differential  response  to  the  task  condition  in  that  the  right 
hemisphere  did  not  show  further  reduction  from  control  to  task 
conditions  as  did  the  right  hemisphere  responses .  The  MEG 
showed  reduction  in  both  hemispheres  to  the  task  condition . 

Dr  Miyamoto  confirmed  the  presence  of  anterior  MEG 
responses  by  locating  magnetic  sources  to  visual  probe  stimuli 
in  the  parietal  area  (Miyamoto  &  Wilson,  1991)  .  Using  this 
information,  we  have  now  repeated  the  original , study  while 
recording  MEG  and  EEG  evoked  responses  from  both  the  occipital 


and  parietal  areas.  Our  prediction  that  there  would  be  a 
greater  amplitude  reduction  in  the  MEG  responses  to  the  probe 
stimuli  in  the  right  hemisphere  parietal  area  than  in  the  left 
was  confirmed.  We  could  thereby  locate  the  active  brain  site 
for  performing  the  rotation  task  in  the  spat ially-dominant 
right  parietal  cortex  using  the  MEG  {Oliver  &  Wilson,  1991) . 

Stimulus  Modality  Interactions 

We  are  also  exploring  the  interaction  of  visual  and 
auditory  stimuli.  To  date,  we  have  found  that  auditory  stimuli 
which  occur  simultaneously  with  or  precede  visual  stimuli  cause 
a  reduction  in  the  magnetic  response  to  the  visual  stimuli  as 
recorded  from  the  visual  cortex.  This  occurs  even  though  there 
is  no  recordable  response  to  the  auditory  stimuli  from  the 
visual  cortex.  The  mechanism  for  this  interference  is  not 
clear.  We  are  planning  future  studies  to  determine  the  locus 
and  nature  of  this  interference.  The  application  of  this  would 
be  in  the  area  of  bisensory  information  delivery  to  crew 
members  and  how  to  either  take  advantage  of  it  or  to  reduce  any 
negative  effects  (Busch,  et  al,  1989)  . 

Related  MEG  Work 

Captain  DeRego  has  developed  models  of  source  activity 
involving  multiple  dipoles.  These  models  are  very  useful  as  we 
move  from  the  simple,  and  inadequate,  single  dipole  model 
towards  the  more  realistic  situation  in  which  several  sources 
are  simultaneously  active.  Capt  DeRego  has  also  developed  the 
hardware  and  software  for  a  system  that  measures  each  subject's 
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head  and  then  locates  the  source  of  MEG  activity  using 
spherical  coordinates.  This  device  has  significantly  improved 
our  measurement  capabilities  (DeRego  &  Badeau,  1989) . 

Through  the  mechanism  of  a  Small  Business  Innovative 
Research  (SBIR)  contract  monitored  by  Maj  Badeau,  we  were  able 
to  simultaneously  collect  37  channels  of  MEG  data  using  the  new 
system  available  at  BTI,  Inc.  at  San  Diego,  CA.  To  compare  the 
37-channel  MEG  responses  to  19-channel  EEG  responses  we 
provided  four  subjects  who  could  travel  to  San  Diego  and  also 
participate  in  electrical  evolced  response  collection  in  our 
laboratory.  We  used  a  simple  approach  in  which  two  tones,  600 
Hz  and  1000  Hz,  were  randomly  presented  to  the  subjects.  The 
stimuli  were  the  same  in  all  conditions,  only  the  cognitive 
processing  was  changed  in  order  to  examine  the  effects  of 
increased  processing  on  the  evoked  responses.  The  tasks  were 
(1)  count  each  tone,  (2)  simple  reaction  time  to  each  tone,  (3) 
a  choice  reaction  time  task  where  the  subjects  pressed  one 
button  to  the  600  Hz  tone  and  another  button  to  the  1000  Hz 
tone,  (4)  a  memory  task  where  the  subjects  pressed  one  button 
if  a  stimulus  repeated  itself  on  the  preceding  trial  or  press 
another  button  if  the  stimulus  was  not  a  repeat.  We  have 
collected,  but  not  fully  analyzed,  the  MEG  data  and  have  found 
evidence  of  a  longer  latency,  slower  component  as  the 
processing  is  increased.  We  will  repeat  the  study  with  the  19- 
channel  brain  mapping  system  with  the  same  subjects  and  examine 
the  data  for  similarities  and  differences. 
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NEW  ANALYSIS  TECHNIQUES 


The  measurement  techniques  currently  used  in  human 
electrophysiology  have  some  limitations  in  operational 
environments  and  thereby  are  also  limiting  the  areas  that  we 
can  explore.  We  need  to  explore  alternatives  to  the  evoked 
potential  and  standard  EEC  analysis  techniques  that  are 
currently  used.  They  are  useful,  but  impose  restrictions  upon 
our  thinking  and  theories.  The  ongoing  EEC  is  considered  by 
most  evoked  potential  researchers  and  theoreticians  as  noise. 

It  is  seen  as  being  in  the  way  since  it  hides  the  "true"  signal 
from  us.  As  a  result  many  procedures  are  used  to  eliminate  this 
"noise."  However,  some  researchers  think  this  is  akin  to 
throwing  the  baby  out  with  the  bath  water.  The  currently  used 
evoked  potential  techniques  do  provide  information  and  probably 
can  be  developed  further.  These  procedures  are  especially 
restrictive  in  the  area  of  operational  environments  where  we  do 
not  have  much,  if  any  control,  over  the  situations  that  we  must 
measure  and  therefore  it  is  difficult  to  use  evoked  response 
techniques.  For  example,  probe  stimuli  may  not  be  presented  at 
the  correct  time,  it  is  difficult  to  synchronize  to  task 
stimuli  and  not  enough  stimuli  may  be  present  to  permit 
response  averaging.  We  are  examining  several  new  developments 
to  see  how  useful  they  can  be  to  us.  Specifically,  nonlinear 
analysis,  neural  networks,  and  alpha  suppression. 

Nonlinear  Analysis  of  EEG  Data 

Major  Albert  Badeau  has  been  using  nonlinear  analysis 
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techniques  to  determine  the  correlation  dimension  of  ongoing 
EEG  data  and  its  relationship  to  subject  state.  The  goal  is  to 
see  if  this  technique  can  give  us  better  metrics  of  operator 
state  by  using  on  going  EEG.  Correlation  dimensions  were 
estimated  to  quantify  the  change  in  EEG  signal  dynamics  as 
subjects  performed  increasingly  difficult  tasks.  EEG  was 
recorded  during  eyes  open  resting,  eyes  closed  resting,  and 
two  difficulty  levels  of  a  visual  probability  monitoring  task. 
The  EEG  signal  was  embedded  into  higher  dimensional  structures, 
and  two  dimensional  projections  were  plotted  as  phase 
portraits,  see  Figure  20.  The  results  indicated  qualitative 
differences  across  conditions.  Correlation  dimension  estimates 
increased  for  both  task  conditions  relative  to  the  no  task 
conditions.  This  technique  seems  promising  and  we  plan  to 
continue  to  explore  the  limits  of  EEG  data  using  this  and  other 
procedures . 


Neural  Networks 

We  have  performed  two  projects  using  neural  networks  to 
classify  subject  cognitive  task  on  the  basis  of  evoked 
potential  and  EEG  data.  Working  with  Lt  Col  Morton,  we  used 
neural  networks  to  classify  single  evoked  potential  trials  in 
data  collected  using  the  odd-ball  paradigm.  The  probability  of 
the  frequent  and  rare  trials  were  70%  and  30%,  respectively. 
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Figure  20.  Phase  plots  of  EEG  recorded  during  four 
conditions .  The  first  condition,  top  row,  was  eyes  closed 
resting,  the  second  condition,  second  row,  was  eyes  open 
resting  condition.  The  third  condition  was  low  difficulty 
level  visual  monitoring  task  and  the  fourth  condition  was 
during  a  more  difficult  level  of  the  monitoring  task.  Each 
column  represents  2-D  projection  of  an  additional  dimension. 
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These  data  were  divided  into  two  sets,  a  learning  or  trianing 
set  consisting  of  363  training  trials  and  100  test  trials.  The 
neural  network  was  trained  using  the  learning  set  of  trials  and 
the  test  set  trials  were  classified  using  the  trained  network. 
Several  configurations  of  the  net  were  used  to  achieve  optimum 
performance.  In  addition  to  the  neural  net,  the  same  data  sets 
were  submitted  to  stepwise  linear  discriminant  analysis  for 
classification.  The  percentage  correct  was  essentially 
identical  for  the  two  techniques,  85%  for  the  neural  net  and 
84%  for  the  discriminate  analysis.  The  neural  net  results  are 
encouraging  since  it  was  able  to  perform  as  well  as  the  more 
traditional  statistical  technique. 

Captain  Cretchen  Lizza  conducted  a  study,  as  part  of  her 
master's  thesis,  in  which  she  used  a  neural  network  to  process 
EEC  data  which  were  collected  while  subjects  were  engaged  in 
several  tasks  varying  from  eyes  closed  resting  to  performing 
two  difficulty  levels  of  two  different  cognitive  tasks.  She 
used  segments  of  on  going  EEC  data  that  were  not  pre  processed 
and  again  was  able  to  demonstrate  that  the  neural  net  could 
perform  as  good  as  or  slightly  better  than  discriminant 
analysis.  Much  of  her  efforts  were  spent  finding  the  optimum 
parameters  for  the  neural  net.  This  was  a  very  time-consuming 
task  and  she  had  to  stop  before  exhausting  all  possibilities. 
More  effort  should  be  spent  learning  about  the  neural  nets  so 
that  they  can  be  optimized  for  classifying  EEC  data.  Future 
efforts  should  also  explore  methods  of  preprocessing  the  EEC 
data  prior  to  submission  to  the  neural  net  to  enhance  the 
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results.  We  hope  that  with  further  investigation  this  technique 
will  provide  a  method  of  on-line  monitoring  of  EEG  data  that  is 
a  reliable,  sensitive  method  of  classifying  subject  state. 

Alpha  Suppression 

Alpha  rhythm  block  or  suppression  was  one  of  the  first  EEG 
phenomena  reported  by  Berger  in  the  first  reports  of  human  EEG. 
Activity  in  the  alpha  band  of  the  EEG,  8-13  Hz,  is  prominent 
over  the  back  of  the  head  during  resting  states,  especially  if 
the  eyes  are  closed.  Opening  the  eyes  or  performing' mental 
activity  causes  the  alpha  activity  to  be  reduced  or  eliminated 
which  is  commonly  called  alpha  block.  Alpha  activity  is 
receiving  renewed  interest  by  several  workers  in  the  area  of 
applied  psychophysiology.  One  of  the  approaches  is  to  examine 
the  levels  of  alpha  activity  following  discrete  stimuli 
requiring  cognitive  activity.  Kaufman  and  co-workers  have 
reported  that  the  blocking  of  MEG  alpha  is  related  to  the 
nature  of  the  cognitive  activity.  That  is,  if  the  task  is 
visual,  the  blocking  occurs  in  areas  of  visual  association 
cortex;  further,  the  length  of  the  blocking  is  proportional  to 
the  length  of  the  required  processing  time  as  measured  by 
reaction  times  (Kaufman,  et  al,  1990) .  Pfurtscheller  and  his 
colleagues  have  performed  similar  experiments  using  electrical 
recordings  of  brain  activity  and  have  labeled  these  responses 
event  related  desynchronization  (Pfurtscheller,  1989) . 

We  plan  to  utilize  these  techniques  with  EEG  data  to 
determine  how  much  they  can  tell  us  about  the  brain's  response 
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to  task  difficulty  in  laboratory  tasks.  If  they  are  successful 
in  measuring  differences  in  task  difficulty,  then  we  will  use 
them  in  field  research  to  see  how  well  they  work  in  that  arena. 
With  our  brain  mapping  system,  we  can  determine  the  brain  areas 
of  maximal  response  and  use  those  sites  when  recording  in 
operational  settings. 

PSYCHOPHYSIOLOGICAL  ASSESSMENT  TEST  SYSTEM  (PATS) 

The  PATS  is  almost  completed  and  is  the  newest  version  of 
our  physiological  test  battery  which  is  superior  to  our  older 
system,  the  Neuropsychological  Workload  Test  Battery  (NWTB) . 

The  primary  areas  of  enhancement  are:  user  interface,  added 
tests,  analysis  tools,  data  storage  capabilities,  and  resident 
statistical  analysis  capabilities.  The  user  interface  was 
designed  with  both  the  inexperienced  user  and  the  simulation 
environment  in  mind.  Using  the  graphics  capabilities  of  the 
Macintosh  II  computer,  a  very  extensive  set  of  interface 
screens  have  been  implemented.  They  take  the  user  through  the 
steps  necessary  to  set  up  a  data  collection  session.  This 
includes  specifying  the  individual  tests,  determining  their 
parameters,  and  establishing  the  conditions  which  start  and 
stop  each  test,  see  Figure  21.  The  available  tests  include, 
auditory  and  visual  Sternberg  memory  tasks,  auditory  and  visual 
rare  event  tasks,  auditory  and  visual  continuous  performance 
tasks,  visual  pattern  reversal,  auditory  brain  stem  response,  a 
tracking  task,  a  visual  monitoring  task  and  a  simple  flash. 

We  have  added  a  user-defined  cognitive  task  in  which  the 
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user  can,  within  the  limits  of  the  shell,  design  a  task  using 
visual  and  auditory  stimuli.  This  feature  will  be  very  useful 
to  the  user.  Several  analysis  tools  have  been  added.  These 
include  digital  filtering,  single  trial  evoked  potential 
analysis,  improved  data  editing  and  manipulation. 
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Figure  21.  Sample  PATS  screen  for  establishing  the 
parameters  for  auditory  stimuli  in  the  rare  event  test.  The 
user  quickley  goes  though  several  screens  such  as  this  to 
establish  the  parameters  for  each  test. 

By  implementing  a  laser  disk,  the  user  will  be  able  to 
collect  and  store  up  to  8  hours  of  16  channels  of  analog  and 
additional  digital  data.  Prior  to  the  advent  of  the  laser 
disk,  this  amount  of  data  could  not  be  easily  stored  digitally 
in  a  small  package.  We  have  found  a  very  good  statistical 
package  that  will  be  resident  on  the  PATS.  The  reduced  data 
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will  be  organized  so  that  it  can  be  directly  submitted  to  the 
statistical  package.  This  will  greatly  reduce  the  data 
handling  required,  speed  up  analysis,  and  omit  the  errors  of 
the  extra  data  handling  now  required  to  get  it  from  one  machine 
to  another.  Statistical  analysis  can  now  be  performed  at  the 
site  of  data  collection  enabling  the  user  to  make  on  the  spot 
decisions  about  the  next  data  to  collect  on  the  basis  of  the 
current  results.  The  users  manual  will  provide  the  information 
required  to  operate  the  system. 

The  PATS  has  been  developed  on  the  basis  of  our  experience 
and  others  experience  with  the  NWTB.  It  has  been  designed  to 
meet  the  needs  of  the  novice  and  experienced  user  in  the  field 
(simulator)  and  laboratory  environments.  We  have  spent  a  great 
deal  of  time  on  the  user  interface  and  the  other  enhancements. 

A  number  of  potential  users  have  already  expressed  interest  in 
using  the  PATS.  Preliminary  versions  have  been  demonstrated  at 
the  Human  Factors  Society  and  the  Aerospace  Medicine 
Association  Annual  Meetings. 


FUTURE  DIRECTIONS 

FIELD  RESEARCH 

We  need  a  larger  data  base  of  flight  data  so  that  we  can 
begin  making  generalizations  about  physiological  data.  The 
more  situations  that  we  cover,  the  better  we  will  be  able  to 
see  both  the  commonalties  across  missions  and  the  unique 
qualities  of  individual  missions,  crew  positions,  and  aircraft. 
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This  will  permit  us  to  provide  more  useful  information  to 
designers  and  evaluators  of  new  systems.  This  data  base  will 
also  be  extremely  useful  for  the  evaluation  of  current  systems 
and  solving  problems  that  arise  with  these  current  systems.  We 
should  continue  data  collection  with  routine  training  missions 
with  additional  types  of  aircraft,  but  also  acquire  data 
representing  extremes  such  as  night  in-air  refueling, 
reconnaissance  and  long-haul  transport  missions. 

When  collecting  data  during  operational  missions,  we  are 
restricted  to  the  available  missions  of  the  flying  unit.  By 
careful  selection  of  aircraft  and  mission  type  we  can 
accomplish  a  great  deal.  However,  we  need  to  have  more  control 
over  some  flights  so  that  we  can  test  important  facets  of  our 
techniques.  For  example,  the  sensitivity  of  the  various 
physiological  measures  needs  to  be  determined.  This  requires 
greater  control  over  what  the  pilots  are  doing  than  is  possible 
during  one  typical  operational  mission.  In  order  to  accomplish 
these  goals,  we  will  have  to  utilize  other  aircraft  and  pilots 
in  situations  where  we  can  design  the  missions  with  the  types 
and  levels  of  workload  that  are  required  to  provide  us  with  the 
data  we  need. 

Probe  and  other  evoked  potential,  as  well  as  ongoing  EEG 
analysis,  methods  should  be  used  in  flight  studies  to  ascertain 
their  utility.  The  capability  to  capture  timing  signals  from 
aircraft  displays  and  auditory  systems  should  be  developed  so 
that  time-locked  brain  activity  in  response  to  these  "naturally 
occurring"  stimuli  can  be  used  to  determine  workload  and  state. 
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LABORATORY  RESEARCH 


Since  flight  and  other  Air  Force  functions  are  by  their 
nature  multiple  tasks,  we  need  to  conduct  laboratory  research 
with  multiple  task  paradigms.  Historically,  performance  and 
psychophysiological  laboratory  research  has  used  single  tasks 
or  dual  task  paradigms.  These  provide  a  great  deal  cf 
experimental  control  and  permit  unequivocal  interpretation  of 
the  effects  of  the  variables  manipulated.  Most  real  world  jobs 
involve  multiple  task  situations  due  to  the  nature  of  the  jobs. 
This  presents  a  problem  of  experimental  control  and  analysis; 
but,  if  we  are  to  understand  the  nature  of  these  situations, 
these  issues  must  be  dealt  with.  The  measurement  of  workload  in 
multiple  tasks  was  addressed  in  three  recent  chapters  in  the 
book  Multiple  Task  Performance  edited  by  Diane  Damos  (Wilson 
&  Eggemeier,  1991;  Eggemeier  &  Wilson,  1991;  Eggemeier,  Wilson, 
Kramer  &  Damos,  1991) .  The  reviews  of  the  literature  for  these 
chapters  emphasized  the  need  for  research  with  multiple  tasks 
and  ^he  problems  of  interpolating  from  single  or  dual  task 
results  to  the  multiple  task  situation.  Batteries  such  as  the 
NASA  Multiple  Attribute  Task  Battery  will  be  used  to  provide 
"real  world"  types  of  environments  to  collect  data.  While  not 
a  substitute  for  actual  "on  the  job  data, "  these  batteries  do 
permit  the  initial  statistical  and  theoretical  examination  of 
more  real-world-like  situations.  This  is  necessary  because 
control  over  the  environment  is  needed  to  understand  the 
complex  interactions  of  events  that  take  place  in  the 
operational  environment.  The  use  of  multiple  task  situations 
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will  require  rethinking  of  our  assumptions  and  methodologies  of 
research.  We  no  longer  have  absolute  control  as  we  did  in  the 
single  task  situation.  This  brings  with  it  many  problems,  but 
problems  that  must  be  faced  if  we  are  to  advance  to  the  applied 
world.  Some  see  this  as  an  impossible  situation,  others  see  it 
as  a  challenge  to  our  creativity. 

With  the  advances  in  imaging  technology,  it  is  now  possible 
to  study  the  spatial  distribution  of  electrical,  magnetic  and 
metabolic  activity  in  the  intact  human  brain.  This  permits  us 
to  study  the  spatial  as  well  as  temporal  pattern  of  information 
flow  in  the  brain.  By  combining  these  imaging  techniques  and 
by  utilizing  the  best  one  for  a  given  problem,  we  are  on  the 
threshold  of  being  able  to  make  large  advances  in  understanding 
how  the  brain  processes  information.  By  developing  functional 
brain  atlas,  we  will  better  understand  how  the  human  brain 
functions  when  it  is  engaged  in  cognitive  activity.  Not  only 
will  we  be  able  to  establish  which  areas  of  the  brain  are  used 
for  different  cognitive  functions,  but  we  will  be  able  to  map 
the  temporal  pattern  of  activity  associated  with  different 
tasks  and  cognitive  functions.  We  can  use  this  knowledge  to 
design  better  displays  and  to  monitor  workload  and  operator 
state.  Having  the  capability  to  select  from  these  very 
powerful  imaging  techniques  would  permit  the  rapid  pursuit  of 
our  goal  of  better  understanding  of  brain  information 
processing  and  the  application  of  this  information  to  Air  Force 
needs . 

Single  and  few  trial  evoked  potential  techniques  should  be 
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explored  further.  Our  airborne  studies  have  shown  that  brain 
evoked  potentials  can  be  recorded  from  pilots.  Better  methods 
of  collecting  and  processing  this  data  are  required  so  that 
this  technique  can  be  applied  in  more  situations.  We  are  also 
working  on  a  device  to  process  cardiac,  respiration,  eye  blink, 
evoked  potential  and  EEC  spectral  data  in  near  real-time.  The 
device  will  detect  the  relevant  parameters  from  the  raw  data, 
such  as  cardiac  R-waves,  and  will  provide  this  information  to 
an  on-line  classifier  that  will  classify  subject  workload  or 
state  and  will  give  this  information  to  the  user  or  system 
being  operated. 

One  component  of  the  advancement  of  workload  and  subject 
state  measurement  is  to  standardize  methodologies.  This  was 
the  goal  of  the  AGARD  Working  Group  on  "Human  Performance 
Assessment  Methods."  This  group  established  the  AGARD  Standard 
Tests  for  Research  with  Environmental  Stressors  (STRES)  battery 
as  a  standardized  set  of  seven  tests  that  could  be  used  by 
researchers  in  the  NATO  countries  and  established  a  data  base 
for  data  exchange  using  this  battery  (Farmer,  et  al,  1989) . 
CSERIAC  and  Ergodata,  in  the  United  States  and  France, 
respectively,  are  repositories  of  this  data  base.  At  our 
instigation,  a  similar  working  group  has  been  established  by 
AGARD  to  standardize  psychophysiological  measures.  This  will 
stimulate  more  laboratories  to  use  these  methods  and  will 
facilitate  data  exchange  by  providing  standard  recording  and 
analysis  techniques. 
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